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A graphical framework is used for statistical analysis of the results from an extensive In- 
version test of a collection of Reynolds-averaged Navier-Stokes computational fluid 
dynamics codes. The solutions were obtained by code developers and users from the U.S., 
Europe, Asia, and Russia using a variety of grid systems and turbulence models for the June 
2009 4 th Drag Prediction Workshop sponsored by the AIAA Applied Aerodynamics 
Technical Committee. The aerodynamic configuration for this workshop was a new subsonic 
transport model, the Common Research Model, designed using a modern approach for the 
wing and included a horizontal tail. The fourth workshop focused on the prediction of both 
absolute and incremental drag levels for wing-body and wing-body-horizontal tail 
configurations. This work continues the statistical analysis begun in the earlier workshops 
and compares the results from the grid convergence study of the most recent workshop with 
earlier workshops using the statistical framework. 
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Nomenclature 

aspect ratio 
drag coefficient 
pressure drag coefficient 
skin friction drag coefficient 
lift coefficient 

pitching moment coefficient 
coverage factor for individual values 
Mach number 

number of observations in a sample 
Reynolds number based on reference chord 

value of an observation 
sample mean of a set of observations 
sample median 
population mean 

estimate of the population mean 

population standard deviation 

estimate of the population standard deviation 

/v , a 

Coefficient of variation = O / jA 


I. Introduction 

I N June 2009, the AIAA Applied Aerodynamics Technical Committee (APATC) sponsored the fourth Drag 
Prediction Workshop (DPW-IV) for transonic cruise drag predictions of subsonic transports. The workshop was a 
follow-on to the first Drag Prediction Workshop (DPW-I) held in June 200 1, 1 ’ 2 the second Drag Prediction 
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Workshop (DPW-II) held in June 2003, 3,4 and the third Drag Prediction Workshop (DPW-III) held in June 2006. 5,6 
The objectives for all four workshops were (1) to assess the state-of-the-art computational methods as practical 
aerodynamic tools for aircraft force and moment prediction of industry relevant geometries, (2) to provide an 
impartial forum for evaluating the effectiveness of existing computer codes and modeling techniques using Navier- 
Stokes solvers, and (3) to identify areas needing additional research and development. 

DPW-I solicited CFD predictions of the lift, drag, and pitching moment for a subsonic transport wing-body 
configuration. The DLR-F4 wing-body configuration 7 " 9 was chosen due to public availability of the geometry and 
experimental data from three wind tunnels. Test cases consisted of a single point solution at a fixed value of C L (C L 
= 0.5±0.001), calculation of a drag polar, and an optional calculation of drag rise at constant values of C L . DPW-I 
did not require a grid convergence analysis. A total of 38 solutions were submitted for the workshop from 18 authors 
using 13 different CFD codes. A summary of the results of DPW-I is given in Ref. 1, and a statistical analysis of the 
results is given in Ref. 2. The code-to-code statistical analysis of the DPW-I results identified two major surprises: 2 * 
(1) roughly 20% of the solutions were statistical outliers compared to the others, and (2) the code-to-code scatter for 
drag was more than an order of magnitude larger than desired by airframe designers. 

DPW-II focused on grid refinement studies and the prediction of installed pylon-nacelle drag increments. The 
DLR-F6 10 Wing-Body and Wing-Body-Nacelle-Pylon configurations were chosen for DPW-II since DLR and 
ONERA made data publicly available for this configuration. Test cases consisted of a single point solution at a fixed 
value of C L (C L = 0.5±0.001) for both the DLR-F6 Wing-Body and Wing-Body-Pylon-Nacelle configurations on 
coarse, medium, and fine grids, and a drag polar. Optional test cases included a comparison of tripped and fully 
turbulent solutions and calculation of drag rise at fixed values of C L . A total of 21 solutions were submitted for the 
workshop from 20 authors using 18 different CFD codes. There were 16 solutions that calculated all three grid levels 
for both the DLR-F6 Wing-Body and Wing-Body-Pylon-Nacelle from 15 authors using 15 different CFD codes. A 
summary of the results is given in Ref. 3, and a statistical analysis of the results is given in Ref. 4. The DLR-F6 
configuration had substantial areas of separation at the wing-body juncture and at the wing-pylon juncture. 
Additionally, there was a region of separation at the trailing edge of the wing. The code-to-code statistical analysis 
of the DPW-II results identified four major findings: 4,6 (1) roughly 20% of the grid convergence study solutions 
were statistical outliers compared to the others, (2) the code-to-code scatter for the wing-body configuration on the 
medium grid was significantly reduced compared to DPW-I, (3) the code-to-code scatter was still significantly 
larger than desired by airframe designers, and (4) there was no significant change in code-to-code scatter with 
increasing grid density. 

The panel session discussion at DPW-II identified three suggestions for a third workshop: (1) the large regions of 
separation were a likely culprit for the lack of grid convergence therefore cases should be chosen with minimal 
separation, (2) simpler cases were required to allow for better grid convergence studies and wider participation, and 
(3) blind studies were preferable where experimental data were not available a priori. It was generally agreed that 
continuing studies of the DLR-F6 case were warranted. Vassberg et al. 11 designed a side-of-body fairing to remove 
the wing-body juncture separation and produce attached flow. The fairing was designated the FX2B and the 
configuration was referred to as the DLR-F6-FX2B. An experimental program was initiated at DLR and NASA to 
collect data for the DLR-F6 with and without the FX2B fairing. 12 The data were not collected before the workshop 
and the workshop was run blind. Additionally, two isolated wings, designated DPW-W1 and DPW-W2, were 
designed (see Ref. 5 for a description of this process) for a simple geometry with DPW-W2 a single point 
optimization of DPW-W1. 

DPW-III focused on grid convergence studies and predicted increments for the DLR-F6 wing-body with and 
without the FX2B side-of-body fairing. The Reynolds number was increased to 5 million to minimize the trailing 
edge separation. The choice of Reynolds number was limited by stress analysis of the wind tunnel model. Additional 
test cases included a drag polar at fixed Mach number and an optional Reynolds number scaling study. An optional 
Case 2 grid convergence study included four levels of grid for the DPW-W1 and DPW-W2 isolated wing cases and 
a drag polar at a fixed Mach number. Case 2 specified the angle-of-attack rather than a fixed C L . A summary of the 
DPW-III results is given in Ref. 5, and a statistical analysis is given in Ref. 6. The code-to-code statistical analysis 
of DPW-III identified four major findings: 6 (1) roughly 20% of the solutions for the grid convergence study of the 
Case 1 DLR-F6 wing-body configuration were statistical outliers compared to the others; none of the solutions for 
the Case 2 wing alone were statistical outliers, (2) the code-to-code scatter for Case 1 DLR-F6 wing-body fine grid 
showed no improvement over DPW-II, (3) the code-to-code scatter was still significantly larger than desired by 
airframe designers, and (4) the code-to-code scatter showed a reduction with increasing grid resolution when the 
variation in lift was appropriately accounted for. However, the code-to-code scatter for DPW-III was larger for the 
coarse and medium grids than for DPW-II. 
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A new wing-body-horizontal tail configuration, with and without nacelle-pylons, was developed by the NASA 
Subsonic Fixed Wing (SFW) Aerodynamics Technical Working Group (TWG) in collaboration with the DPW 
Organizing Committee. 13 The Common Research Model (CRM) is representative of a contemporary high- 
performance transonic commercial transport. DPW-IV was a true blind test; the workshop was held in June 2009 
and all the data submittals were completed before experimental data were collected. Experimental data were 
collected in the NASA Langley National Transonic Facility in January-February 2010 and in the NASA Ames 1 1 -ft 
wind tunnel during March- April 20 10. 14 DPW-IV focused on the prediction of both absolute and incremental drag 
levels and included grid convergence studies for the CRM wing-body-horizontal and prediction of trimmed drag. A 
summary of the results from DPW-IV is provided in Ref. 15. Detailed results from participants are provided in Refs. 
16-28. 

This paper is organized in the following manner. Section II provides a description of the statistical analysis. 
Section III outlines the test cases for the workshop. Section IV details the statistical results. Section V provides some 
summary comments. 


II. Statistical Approach 

Hemsch 2 introduced the idea of treating different computations of a test case as a collective and using N- version 
testing in a statistical framework to investigate the submissions. No individual result is considered the “right” or 
“best” result. The dispersion of the results is treated as noise in the collective computational process. 

A running record of individual outcomes is plotted for each of the measures of interest and derived quantities 
reported by participants. Participants reported the angle of attack (a), total drag coefficient ( C D ), pressure drag 
coefficient (C Drr ), skin friction drag coefficient ( C D ^ ), and moment coefficient (C m ). The value of the measure 

of interest is plotted on the vertical axis and an integer index is used for each data submission on the horizontal 
index. The order of the solutions on the horizontal axis is irrelevant since this is not a temporal axis, therefore the 
submissions are grouped according to grid type (structured and unstructured). 

An estimate of the population mean fa of the plotted data submissions is made and is shown on the graph as the 
centerline. Upper and lower scatter limits are placed on the graph as follows: 

Upper Limit = fa + Ko ( 1 ) 

Lower Limit = fa - Ko (2) 

where O is an estimate of the population standard deviation and K is an appropriate coverage factor. Significant 
results are outcomes that lie outside the process limits defined in Eqs. (1) and (2). These results, referred to as 
outliers, represent submissions that are different from the results that lie within the scatter limits (and should be 
investigated to understand the difference). 

The population mean fa is estimated using the sample median, which is given (for sorted data) as: 


fa = x 

X = X (n +\)/2 ( n odd ) ( 3 ) 

x = 0.5 (x n/2 + X (n/2)+l ) (n even) 

The sample median provides a robust estimate when outliers are present. The sample standard deviation 


a = SSD » 


1 


-4 2(*,-*> 2 

n- lri 


( 4 ) 


is used to estimate the population standard deviation. The coverage factor is estimated for a uniform distribution 29 as 

K = S. 
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III. DPW-IV Test Cases 

The geometry for DPW-IV was the Common Research Model (CRM) as described in Vassberg et al. 11 and 
shown in Figure 1. Case 1 consisted of a grid convergence study and a downwash study and was required of 
participants. Case 2 Mach Sweep Study and Case 3 Reynolds Number Study were each optional. All simulations 
were specified to be in free air, i.e. no wind tunnel walls or model support systems were to be included. Boundary 
layers were to be modeled as fully turbulent, i.e. no transition location was specified. The moment reference center 
is Xref = 1325.90 in, z REF = 177.95 in. Additional reference quantities are: Sref = 594,720 in 2 ; Cref = 275.80 in; and 
AR = 9.0. 

Grids were made available for the three grid types - multi-block structured, overset structured, unstructured - or 
participants could provide their own grids. A summary of the grids and results is provided in Ref. 15. Additional 
details of individual grid families and results from participants are available in Refs. 16-28. 

A. Case 1 - Grid Convergence and Downwash Studies: 

Case 1 focused on the CRM Wing-Body and Wing-Body-Horizontal tail configurations. The Reynolds number 
was chosen to be 5 million to match what would be available in the experimental campaign: 

1) Grid convergence study at Mach = 0.85, C L = 0.500 (±0.001) 

• Tail incidence angle i H = 0° 

• Coarse, Medium, Fine grids required; Extra-Fine grid optional 

• Chord Reynolds number 5 x 1 0 6 based on Cref 

• Reference Temperature = 100° F 

2) Downwash study at Mach = 0.85 

• Calculate Drag Polars for a = 0.0°, 1.0°, 1.5°, 2.5°, 3.0°, 4.0° 

• Tail incidence angle i H = -2°, 0°, +2°, and Tail off 

• Medium grid 

• Calculate Trimmed Drag Polar (CG at reference center) derived from polars at iH = -2°, 0°, +2° 

• Calculate Delta Drag Polar for tail off vs. tail on, i.e. Wing-Body vs. trimmed Wing-Body-Horizontal 

Tail 

• Chord Reynolds number 5 x 1 0 6 based on Cref 

• Reference Temperature = 100° F 

B. Case 2 (Optional) - Mach Sweep Study: 

Calculate Drag Polars at: 

• Mach = 0.70, 0.75, 0.80, 0.83, 0.85, 0.86, 0.87 

• Drag Rise curves at C L = 0.400, 0.500, 0.500 (±0.001 or extracted from polars) 

• Untrimmed, Tail incidence angle i H = 0° 

• Medium grid 

• Chord Reynolds number 5 x 1 0 6 based on Cref 

• Reference Temperature = 100° F 

C. Case 3 (Optional) - Reynolds Number Study: 

Compare results from Re = 5 x 10 6 with Re = 20 x 10 6 

• Mach = 0.85, CL = 0.500 (±0.001) 

• Tail incidence angle i H = 0° 

• Medium grid 

• Chord Reynolds number based on Cref 

• For Re = 20 x 1 0 6 , Reference Temperature = -250° F 


IV. Results 

The data was submitted to the DPW committee before the workshop. After the workshop, authors were given 
time to evaluate and resubmit their solutions. Several authors took advantage of this time and submitted corrections 
to their solutions or provided additional data that they did not have time to complete before the workshop. All data 
submissions were stopped before the CRM model entered the wind tunnel test program. 
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A. Case 1 Grid Convergence Study 

For the Case 1 Grid Convergence Study: 

1 . There were 1 6 codes used: 6 structured grid codes, 1 overset structured grid code, and 9 unstructured grid 

codes (8 mixed element based on tetrahedral, pyramids, and prisms; 1 unstructured hexahedral code). 

2. There were 29 solutions computed: 9 structured grid solutions, 1 overset structured grid solution, and 19 

unstructured grid solutions (17 mixed element based on tetrahedral and prisms; 2 unstructured 
hexahedral based grids). 

3. There were 5 turbulence models used (not counting variants of basic models): the one-equation Spalart- 

Allmaras model; the two-equation SST k-oo, k-8, explicit algebraic Reynolds stress model (EARSM) 
models; and a Reynolds stress closure using a blended SSG/LRR pres sure -strain model. Some authors 
reported variants of the basic models that they used. 

4. Only 24 submissions included all three required grid levels - Coarse, Medium, and Fine - and are termed 

the nested solutions. 

5. Twelve submissions included the optional Extra-Fine grid. 

The statistical analysis below identifies outliers that are different from the other solutions. The core solutions are 
the solutions that are similar to each other, i.e. the core solutions are the nested solutions minus the outliers. Table 1 
summarizes the number of submissions, nested submissions, and core submissions for DPW-II, DPW-III, and DPW- 
IV. The number of submissions and the number of solutions in the core is larger for DPW-IV than either DPW-II or 
DPW-III. The number of authors in the core is higher for DPW-IV than DPW-II or DPW-III and the number of 
institutions and codes is higher than DPW-III and the same as DPW-II. 

Table 1. Comparison of workshop statistics for grid convergence case in DPW-II, DPW-III, and DPW-IV. 



DPW-II 

DPW-III 

DPW-IV 

Submissions 

Nested 

Core 

Submissions 

Nested 

Core 

Submissions 

Nested 

Core 

Solutions 

21 

16 

13 

26 

20 

16 

29 

24 

21 

Authors 

20 

15 

12 

15 

12 

9 

17 

16 

13 

Institutions 

16 

14 

11 

13 

10 

9 

15 

14 

11 

Codes 

18 

15 

12 

14 

12 

8 

16 

15 

12 


7. Individuals Charts for Case 1 Grid Convergence Study 

Running records of individual outcomes were plotted for each of the measures of interest for each grid level in 
the grid resolution study. The population mean, estimated by the median calculated from Eq. (3), is shown as a solid 
black line and the scatter limits calculated from Eq. (1) are shown as dashed black lines. In all of the following plots 
for Case 1, the solution index on the abscissa refers to the same solution submission and was randomly assigned. 
Figures 2-4 show the 5 measures of interest for the Coarse, Medium, and Fine grid levels, respectively. 

Two solutions are outside the limits for the total drag (Figs. 2a, 3a, 4a) and the pressure drag (Figs. 2b, 3b, 4b) 
for most of the grid levels: Solutions 7 and 24. Only one solution, Solution 13, is slightly below the limits for skin 
friction drag (Figs. 2c, 3c, 4c) on the Coarse and Medium grid levels. 

For angle of attack, solution 18 is slightly outside the limits for the Coarse grid (Fig. 2d), solution 14 is well 
outside the limits on the Medium grid (Fig. 3d), and solution 9 is well outside the limits on the Fine grid (Fig. 4d). 
All of these solutions are within the limits on all of the other grid levels. All of these solutions should check the level 
of iterative convergence for these cases as the angle of attack lies well outside the limits of the collective at only one 
location for the three separate solutions. 

For the pitching moment coefficient, solution 21 is consistently outside the limits on all three grid levels (Figs. 
2e, 3e, 4e). 

2. Graphical Evaluation of Grid Convergence for Case 1 

In order to compare results on unstructured meshes with results on multi-block structured meshes and overset 
structured meshes, it is convenient to use a one-dimensional estimate of the grid spacing which is given for three 
dimensions as 
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1 


Mnpts 

where NPTS is the number of solution points (for either cell-centered or vertex based CFD codes) in the mesh. 
Second order accurate results in the asymptotic range will lie on a straight line when plotted versus h 2 (NPTS" 2/3 ). 
An additional benefit of plotting versus NPTS" 273 is that coarse meshes show up on the right of the plot and finer 
grids approach the y-axis, which is the limit of infinite grid resolution. 

Salas 30,31 has shown that the error is linear with If (where p is the order of accuracy of the scheme) only if the 
grid is refined uniformly in all coordinate directions, e.g. if the grid spacing is halved in the x coordinate, it must 
also be halved in the y and z coordinates. The gridding guidelines posted to guide development of grids for the 
workshop ( http://aaac.larc.nasa.gov/tsab/cfdlarc/aiaa-dpw/Workshop4/gridding guidelines_4.html T required that the 
grid convergence cases maintain the same family of grids between grid levels. However, it was not verified that all 
submitted grids met this requirement. Other factors can also contribute to this non-linearity. 

It is difficult to see the grid related trends in the submissions looking at the running records. Therefore the total 
drag and the moment coefficient for all of the submissions for Case 1, including the partial submissions that are not 
part of the nested data and the Extra Fine grid data which were not used to calculate statistics, are plotted in Figs. 5 
and 6 (one of the solutions was deleted from the pitching moment coefficient plot since it was a factor of 10 larger 
than any other result and was only a partial submission and not included in the statistics). These figures include the 
estimate of the mean shown as a solid black line and the limits shown as dotted black lines. Structured grid 
solutions, including the overset solution, are plotted with green lines and symbols and the unstructured solutions are 
plotted with blue lines and symbols. Solutions 7, 21, and 24 fall outside the limits of the solutions. 

3. Analysis of Outliers 

The previous two sub-sections identified Solutions 7, 21, and 24 as outliers. 

Solution 7 

Solution 7 is significantly outside the limits for the Coarse and Medium grids for both total and pressure drag 
and is outside the limits for the Fine grid also for both total and pressure drag. However, solution 7 is closer to the 
limits and the mean for both total and pressure drag as the grid is refined. The skin friction drag for solution 7 is 
inside the limits for all grid levels. The skin friction drag for solution 7 also starts high relative to the mean and gets 
closer to the mean as the grid is refined. 

Core solution 2 is within the limits for all of the measures of interest. Solution 2 and solution 7 are both multi- 
block structured solvers, both used grids that were generated using the ICEM/Hexa grid generator, and both used the 
SST k-oo turbulence model. The solution 7 grids had more grid points on all grid levels than the solution 2 grids. 
Solution 7 reports using a 4 th order central differencing scheme and solution 2 used a 2 nd order upwind scheme. 
There are several reasons that could explain the differences between solution 2 and 7 : 

1. Numerical scheme: the truncation error of the 4 th order and the 2 nd order scheme will be quite different. 

The grids could still be coarse enough that the cross-over point where the truncation error of the higher- 
order scheme overtakes the lower order scheme has not been reached on the Coarse and Medium grids. 
This could be tested by comparing the results of the two codes on a simpler case in a solution 
verification study. 

2. Lack of iterative convergence: the solutions might not have been iterated sufficiently to reach a steady 

solution. Flows with separation are notoriously slow to converge. This could be tested by running 
additional iterations and monitoring the convergence of the measures of interest. 

3. Grid: while the number of grid points for solution 7 is greater than the number of grid points in solution 2 

grids, the distribution of the grid points could be quite different. Solution 2 could have provided 
resolution for flow features that affect the measures of interest closer to what the other participants 
used. This could be tested by evaluating the grids and running the grids from solution 2 with the code 
from solution 7 and vice versa. 

4. Turbulence model implementation: while both authors report the same turbulence model, experience has 

shown that implementations of turbulence models often require limiters or fixes to achieve robust 
behavior. Additionally, there are several variants of the model reported in the literature and each author 
could be using a different variant. This could be tested by running a turbulence model verification study 
or using the Method of Manufactured solutions for the code with the turbulence model used in these 
solutions. 
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Solution 24 

Solution 24 is well outside the limits (high) for pressure drag on the Medium and Fine grids (Figs. 3b, 4b). 
Solution 24 is also outside the limits (high) for total drag on the Medium and Fine grids (Figs. 3a, 4a). Solution 24 is 
at or slightly below the lower limit for the skin friction drag (Figs. 2c, 3c, 4c) which helps to offset the pressure drag 
and make the total drag closer to the limits. However, the trend with grid resolution is in the wrong direction: the 
solutions get worse relative to the collective as the grid is refined. The number of grid points used for solution 24 is 
on the low side of the mean for each of the grid levels, but there are multiple solutions that use fewer grid points 
than solution 24 and are within the limits for all of the grid levels. Solution 24 used the Spalart-Allmaras turbulence 
model that many other authors used. However, solution 24 used a grid system that no other group used so it is 
difficult to draw further conclusions. 

The distribution of the grid should be checked since the behavior as the grid is converged is different from the 
collective. The lack of iterative convergence and turbulence model implementation listed above should also be 
investigated. 

Solution 21 

Solution 21 is considerably low for the pitching moment coefficient for all three grid levels. However, all other 
measures of interest (angle-of-attack, total drag, pressure drag, skin friction drag) are well within the limits of the 
collective. Solution 21 used the Spalart-Allmaras model used by many other authors that were within the limits of 
the collective. Solution 21 used a hybrid unstructured mesh that was not used by anyone else so it is difficult to draw 
additional conclusions. It seems unlikely that a minor adjustment of a turbulence model would make a large change 
to the pitching moment and leave the drag unchanged. 

There are several reasons that could explain this behavior: 

1. Error: this could be a simple error in the input of the reference moment center or the non- 

dimensionalization of the moment. 

2. Lack of iterative convergence: moments typically require more iterations to converge than forces. 

However, the size of the discrepancy in the moment makes this possibility unlikely. 

4. Core Solutions 

Since solutions 7, 21, and 24 exhibit substantially different behavior from the rest of the collective over a range 
of grids, not just on a single grid, they are removed from the collective and are not part of the core solutions. The 
core solutions consist of the other 21 solutions. 

The median values for the Case 1 Grid Convergence Study core solutions at three grid levels is given in Table 2. 
Table 3 shows the estimate of the standard deviations for the core solutions for the Case 1 Grid Convergence Study. 

Figures 7 through 10 show the variation of the core solutions including new estimates of the median and scatter 
limits for the total drag, pressure drag, skin friction drag, and the pitching moment coefficients, respectively. These 
figures include the estimate of the mean for the core solutions shown as a solid black line and the limits of the core 
solutions shown as dotted black lines. Structured grid solutions, including the overset solution, are plotted with 
green lines and symbols and the unstructured solutions are plotted with blue lines and symbols. The core solutions 
for the total drag and drag components show a converging trend as the grid is refined and are well delimited by the 
limits. However, the pitching moment shown in Fig. 10 shows a divergent trend with grid refinement. 


Table 2. Comparison of Case 1 Grid Convergence Study core medians. 



Coarse 

Medium 

Fine 

a, degrees 

2.340 

2.324 

2.343 

C D , counts 

279 

274 

272 

C D pr, counts 

158 

151 

148 

Cdsf, counts 

124 

125 

125 

Cm 

-0.0425 

-0.0408 

-0.0398 


7 

American Institute of Aeronautics and Astronautics 




Table 3. Comparison of Case 1 Grid Convergence Study core standard deviations. 



Coarse 

Medium 

Fine 

a, degrees 

0.085 

0.084 

0.24 

C D , counts 

8.8 

6.1 

6.1 

C D pr, counts 

8.8 

4.8 

4.6 

Cdsf, counts 

5.7 

4.5 

3.9 

Cm 

0.0059 

0.0060 

0.0068 


B. Comparison of DPW-IV results with DPW-II and DPW-III 

DPW-II, DPW-III, and DPW-IV each included a grid convergence study. Gridding guidelines were posted 
before each workshop and included recommended sizes for each of the grids in the grid convergence studies to 
provide a guide to the participants. Table 4 lists the recommended grid sizes (number of solution points) for the 
three workshop grid convergence studies that were fixed C L , along with the minimum and maximum grid sizes from 
the participants. The grid sizes have tripled over the course of these three workshops. Some participants submitted 
solutions on grids that had half the solution points that the gridding guidelines requested, and some participants 
provided solutions on grids that had over twice the solution points that the gridding guidelines requested. However, 
the questions remains whether increasing grid sizes have improved the results. 


Table 4. Millions of solution points for DPW-II, DPW-III, and DPW-III grid convergence studies. 



DPW-II 

DLR-F6 

Wing-Body 

DPW-III 

DLR-F6 with FX2B fairing 
Wing-Body 

DPW-IV 

CRM 

Wing-Body-Horizontal Tail 


Coarse 

Medium 

Fine 

Coarse 

Medium 

Fine 

Coarse 

Medium 

Fine 

Guidelines 

1.3 

4.0 

12.0 

2.7 

8.0 

24.0 

3.5 

10.0 

35.0 

Minimum 

0.99 

1.8 

2.4 

2.3 

4.7 

9.8 

3.2 

9.9 

25.7 

Maximum 

5.5 

7.9 

23.2 

5.6 

14.6 

41.1 

11.5 

21.3 

58.1 


The coefficient of variation, 32 C v (= a/p), provides a measure to compare the variation of populations with 
different means. The results of DPW-IV are compared to the results of DPW-II and DPW-III 6 for the cases that were 
run at constant lift. Morrison and Hemsch 6 showed that the variation in lift affected the comparisons of drag 
quantities and pitching moment and showed that quantities that corrected to a constant lift condition, e.g. L/D and 
idealized profile drag, were required to compare results at different values of lift. 

The coefficient of variation for the pitching moment coefficient would be negative since the mean is negative 
and the estimate of the standard deviation is positive. Therefore, \C V \ is plotted for the pitching moment coefficient to 
make comparisons easier. 

The DPW-IV results for the total drag, pressure drag, and skin friction drag coefficients, shown in Figs. 11 
through 13 respectively, are very close to the results for the DLR-F6 FX2B attached flow case studied in DPW-III 
that included a fairing that was designed to clean up the side-of-body separated flow. The DPW-III results without 
the fairing (noted on the plots as DPW-III DLR-F6) typically show larger variation most likely due to the separation 
regions in the flow around this configuration. 

However, the C v of the pitching moment coefficient, shown in Fig. 14, has twice the relative variation of the 
DPW-II and DPW-III results on the coarse mesh and increases to approximately four times the relative variation on 
the finest meshes. The variation in the pitching moment coefficient shows a converging trend for DPW-II and DPW- 
III but shows a clear diverging trend for DPW-IV. 
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What is the cause of the large relative variation in the pitching moment coefficient when the drag coefficients 
show very similar trends as previous workshops? The data collection forms for the workshop had an optional entry 
to collect forces and moments on the wing, fuselage, and tail components of the CRM configuration. Twelve 
participants provided these component forces and moments. Only 10 of the component contributions were part of 
the core solutions. Statistics were calculated on these 10 core solutions for each of the components using the 
approach given above. The median was used to estimate the mean and was compared to the arithmetic mean; these 
values were in good agreement. The medians of the lift, drag, and pitching moment are given in Table 5 for the 
wing, fuselage, and tail components for the coarse, medium, and fine grids. The median of the pitching moment 
coefficient, shown in Fig. 15, changes very little with increasing grid refinement; the largest change in the pitching 
moment median is on the tail with approximately a 2.8% change from coarse to fine grid; the change on the fuselage 
is approximately 1.0% and the change on the wing is less than 1%. The lift coefficient has less than 1% change from 
coarse to fine grid on the wing and fuselage, but has approximately a 2.8% change with increasing grid resolution on 
the tail. The drag coefficient has a 21% change on the tail from coarse to fine grid; the change in the drag coefficient 
on the wing is approximately 1.2% and on the fuselage is approximately 2.9%. The horizontal tail contribution to the 
pitching moment is about 50% larger than the fuselage contribution to the moment. 

Table 5. Comparison of component medians for lift, drag, and pitching moment. 



Wing 

Fuselage 

Horizontal Tail 


Coarse 

Medium 

Fine 

Coarse 

Medium 

Fine 

Coarse 

Medium 

Fine 

C L 

0.456 

0.456 

0.457 

0.0615 

0.0613 

0.0612 

-0.0171 

-0.0175 

-0.0176 

C D , counts 

164 

162 

162 

107 

105 

104 

9.75 

8.45 

8.05 

c m 

-0.157 

-0.157 

-0.156 

0.0472 

0.0472 

0.0477 

0.0658 

0.0671 

0.0677 


The standard deviation for each component is given in Table 6 and shown in Figs. 16 through 18. The code-to- 
code scatter for the lift coefficient, Fig. 16, on the horizontal tail is almost as large as the scatter on the wing even 
though the magnitude of the lift coefficient on the wing is approximately 26 times that of the tail. The lift coefficient 
scatter on the fuselage is substantially smaller than the wing or tail. The code-to-code scatter on the drag coefficient, 
Fig. 17, is larger on the fuselage than on the wing for all grid levels; the scatter for the drag on the finest grid is 
largest on the fuselage, almost the same size for the horizontal tail, and smallest on the wing. The code-to-code 
scatter in the pitching moment coefficient, Fig. 18, on the horizontal tail is larger than the code-to-code scatter for 
pitching moment on the wing; the scatter on the fuselage is substantially smaller than the tail or wing. 

Table 6. Comparison of component standard deviations for lift, drag, and pitching moment. 



Wing 

Fuselage 

Horizontal Tail 


Coarse 

Medium 

Fine 

Coarse 

Medium 

Fine 

Coarse 

Medium 

Fine 

C L 

0.0021 

0.0023 

0.0016 

0.00065 

0.00061 

0.00054 

0.0017 

0.0019 

0.0015 

C D , counts 

2.7 

1.1 

1.0 

3.0 

1.9 

1.5 

1.0 

1.3 

1.3 

c m 

0.0053 

0.0045 

0.0039 

0.0014 

0.0012 

0.0018 

0.0067 

0.0076 

0.0058 


This component data gives us insight in the modeling of the wing, fuselage, and horizontal tail; the code-to-code 
scatter on the horizontal tail is nearly as large as either the wing or fuselage even though the median on the 
horizontal tail is substantially smaller than the other two components. Figure 19 shows the coefficient of variation 
for the wing, fuselage, and horizontal tail for the lift, drag, and pitching moment (note that the case was run as fixed 
total lift). The relative variation for the tail is at least twice the relative variation of the fuselage or the wing for lift, 
drag, and pitching moment. The relative variation on the fuselage and wing are more similar to each other. The 
relative variation in the drag on the tail is even larger than the relative variation for the lift or pitching moment. The 
large relative variation in the forces and moments on the tail demonstrate that the solutions are not modeling the 
flow in the tail region as well as the flow on the wing and fuselage. There are several potential causes for the large 
relative variation on the tail: 
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1 . Errors in defining the force and moment integration domains for all of the components 

2. Differences in grid resolution; the grid is better resolved for the wing as the grids are refined than for either 

the fuselage or the tail 

3. Differences in wake resolution of the flow from the wing impinging on the tail; this will lead to different 

local flow angles-of-attack for the tail for different solutions 

4. Effects of turbulence modeling on the tail 

5. Effects of transition location on the tail 

6. Separation on the tail. 


V. Concluding Remarks 

A statistical analysis applied to the results of the 4 th Drag Prediction Workshop demonstrated that the code-to- 
code scatter of the total drag, pressure drag, and skin friction drag decreased with increasing grid resolution. The 
relative code-to-code scatter was compared to the results from the second and third DPW and showed comparable 
levels. However, the code-to-code scatter in the pitching moment showed a divergent trend as it increased with 
increasing grid resolution. Several reasons why the pitching moment would show more scatter for the wing-body- 
horizontal tail configuration were identified. Additional analysis will be required to confirm the cause. 

The statistical analysis identified approximately 12.5% of the solutions that were outliers compared to the other 
solutions. Surprisingly, this has changed by only a small amount from the first workshop through the fourth 
workshop (this reduction corresponds to 1-2 less outliers out of 24 solutions for DPW-IV compared to the first three 
workshops). Some of these outliers have been due to human error and some due to code-to-code bias. Continuing 
efforts to improve CFD solution processes are required to address human errors. Continuing efforts are required to 
understand code-to-code bias. 

The level of code-to-code scatter remains about the same for the second through the fourth workshop. The level 
of scatter has not been reduced even though the grids have approximately tripled over the three workshops. This 
level of scatter is still substantially above the level desired by designers. 

Some recommendations for future workshops: 

1 . Analyze grids (check resolution of the grids on the fuselage and tail in addition to the wing, verify that grids 

meet the gridding guidelines, verify that grids are a family, evaluate grid requirements for codes on simpler 
problems, etc.); grids have been a large source of uncertainty in all four workshops. 

2. Report iterative convergence results 

3. Use common nomenclature 33 when reporting the turbulence model 

4. Do verification tests of turbulence models 33 before applying them to complex flows. 
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Figure 1. Common Research Model geometry (wing-body-horizontal tail). 
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Figure 2. Coarse grid measures of interest for Case 1 Grid Convergence Study. 
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a) Medium grid total drag coefficient 



b) Medium grid pressure drag coefficient 




d) Medium grid angle of attack 



c) Medium grid skin friction drag coefficient 

Figure 3. Medium grid measures of interest for Case 1 Grid Convergence Study. 
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a) Fine grid total drag coefficient 



b) Fine grid pressure drag coefficient 



d) Fine grid angle of attack 




c) Fine grid skin friction drag coefficient 


Figure 4. Fine grid measures of interest for Case 1 Grid Convergence Study. 
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Figure 5. Grid convergence of total drag coefficient including estimate of the mean and limits for all 
Case 1 submissions. 



Figure 6. Grid convergence of pitching moment coefficient including estimate of the mean and limits for all 
Case 1 submissions. 
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Figure 7. Grid convergence of total drag coefficient including estimate of the mean and limits for Core 
submissions for Case 1. 
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Figure 8. Grid convergence of pressure drag coefficient including estimate of the mean and limits for Core 
submissions for Case 1. 
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Figure 9. Grid convergence of skin friction drag coefficient including estimate of the mean and limits for 
Core submissions for Case 1. 



Figure 10. Grid convergence of pitching moment coefficient including estimate of the mean and limits for 
Core submissions for Case 1. 
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Figure 11. Coefficient of variation of the total drag coefficient. 
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Figure 12. Coefficient of variation of the pressure drag coefficient. 
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Figure 13. Coefficient of variation of the skin friction drag coefficient. 
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Figure 14. Coefficient of variation of the pitching moment coefficient. 
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Figure 15. Effect of grid convergence on the pitching moment coefficient for each element of the CRM (wing, 
horizontal tail, fuselage). 



Figure 16. Effect of grid convergence on the standard deviation of the lift coefficient for each element of the 
CRM (wing, horizontal tail, fuselage). 
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Figure 17. Effect of grid convergence on the standard deviation of the drag coefficient for each element of the 
CRM (wing, horizontal tail, fuselage). 
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Figure 18. Effect of grid convergence on the standard deviation of the pitching moment coefficient for each 
element of the CRM (wing, horizontal tail, fuselage). 


22 

American Institute of Aeronautics and Astronautics 



0.20 


0.15 


U 0.10 ■ 


0.05 


0.00 

0.E+00 2.E-05 4.E-05 6.E-05 


X— 



£ 


=^L 


— CL_Wing 
-X- “ CD Wing 
— ♦— CM_Wing 
— CL_Tail 
— X- “CD Tail 
"♦"CM_Tail 
■ CLFus 
-X- -CD_Fus 
-“♦-'CM Fus 


NPTS' M 

Figure 19. Coefficient of variation of lift, drag, and pitching moment coefficients by component of CRM. 
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